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Abstract — An online discussion board called the Collab21earn Discussion Board (DB) 
enables participants' to learn collaboratively within groups. Groups are limited in using 
different feedback formats; either textual or iconic. 57 participants from an institute of 
higher learning tested Collab21earn DB usability. System Usability Scale (SUS) was used to 
measure Collab21earn DB usability. Its average scores were compared between two 
different feedback formats (textual and iconic). Results indicated Collab21earn DB is a 
usable online discussion board however further improvements need to be done especially in 
the prototype's functionalities integration and consistency. 

Index Terms — Usability, online discussion board, System Usability Scale (SUS), 
Collab21earn Discussion Board 

I. Introduction 

Computer applications via networking such as Curriculum Information Document Online System (CIDOS; 
http://www.cidos.edu.my) which was developed by Department of Polytechnic Education aims to support 
teaching and learning for polytechnics in Malaysia since the year 2010. Online discussion board application 
existed in CIDOS or any other online learning systems allow discussions, sharing knowledge and 
communication between learners and instructors possible at flexible time via online. A prototype called 
Collab21earn Discussion Board (DB) was developed to support current online learning environment and also 
to promote collaborative learning between learners within specified group. Collab21earn DB allows learners 
to discuss problems or issues related to workplace scenarios through online. Improving Collab21earn DB is 
important to ensure it is usable for learners to discuss collaboratively within their groups at specified 
functionalities. 

Pertaining to the purpose of Collab21earn DB development, this study is focused with Collab21earn DB 
usability at different feedback format (textual and iconic). 

II. Literature Review 

A. Collaborative Learning Using Online Discussion Board Overview 

Students interacting, working together and participate in group discussion conducting the same tasks to 
achieve common goal can be described as collaborative learning [1]. Collaborative learning implemented 
through online application such as discussion board, chat tool and social networking sites (eg. Facebook, 
Twitter, MySpace) is popular among learning institutes including higher learning institutes in many 
countries. Collaborative learning using online discussion board allow learners to not just interact 
academically with their classmates and instructors but also conduct sociocultural interactions with others 
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outside their learning institutes [2]. 

Nicenet Internet Classroom Assistant (http://www.nicenet.org/ICA/login.cfm) and ATutor Learning 
Management Tool (http://atutor.ca/) are example of online discussion board available in the internet focusing 
on the teaching and learning. Facebook (http://www.facebook.com) and Twitter (https://twitter.com/) are 
example of Social Networking Sites subscribed by users through the internet while phpBB Free Open Source 
Bulletin Board Software (https://www.phpbb.com/) is one of application to create an online bulletin board for 
any type of discussion. These online applications are accessible by users to communicate and interact with 
others in their flexible time as long there are network. 

Online discussion board has been implemented in various learning institute to allow learners communicate 
and interact with others. Facebook and Twitter allows user to communicate and interact with others, however 
with various functionalities embedded in the social networking sites, it is hard to examine specific tasks 
within a controlled time thus easily maneuvered students concentration on specific problem. phpBB is 
cumbersome to apply because requires developers to reset for specific purpose and on specific tasks before 
being upload to a rental web server. Applications developed intelligent systems specific to their own 
technical subjects. Restricted installation based on personal computer requires technical person support in 
managing the system. 

Currently, there is no online discussion board discussing on workplace scenarios implemented in 
polytechnics or community colleges in Malaysia. Curriculum Information Document Online System 
(CIDOS) allows students to share documents related to learning topics using multi format quiz (list box, 
check box (multiple choice or true/false) for assessment purposes, chat tool for synchronous communication 
and multithread forum for asynchronous communication [3]. Similar multithread forum is applied for e- 
learning at Universiti Science Malaysia (elearning@USM, http://www.elearrning.usm.my) whereby students 
are allowed to communicate with others and feedback is given in written format. 

Lack of online discussion board developed with both asynchronous and synchronous communication type 
that allows reflective activity based on feedback to group members' contribution motivates the study to 
develop a prototype for online interaction and communication between students. 

An online discussion board allows learners to share knowledge, participate in learning discussions and 
interact with others at their own time or given specific time [4 and 5] through online or networks. Comments, 
feedbacks and response from the instructor or other learners contribute to the learners understanding on 
specific issues. Collaborative learners would achieve certain understanding of knowledge from workable or 
usable online discussion board. This study focused to the usability of a developed online discussion board 
called Collab21earn Discussion Board. 

B. Protytype Design and Development 

i. Scope of Prototype 

Proposed system called Collab21earn Discussion Board will support collaborative learning within online 
environment. Collab21earn Discussion Board is referred to 'Collaborative learning using discussion board'. 
The scope of Collab21earn DB is as below: 

• System functionality: Collab21earn DB is divided in two (2) type of users; end users and administrator. 
End users are allowed to participate in online discussion and able to use only specific functions controlled 
by the administrator. Administrator has the control to register users, create groups and learning content; 
and control learning session period. This paper only explained end users functionalities and screen shots 
in Collab21earn Functionalities section. 

• Prototype accessibility: Collab21earn DB is accessible through the network at 
http://www.collab21earn.net. However, only registered users within specific time are able to login 
Collab21earn. 

• Questionnaire module: Questionnaires are displayed at the end of learning session for end users to answer 
items evaluating dependent variables. Administrator has the ability to view questionnaire summary 
answered by end users. 

ii. Collab21earn Functionalities 

End users or respondents are divided in groups created by administrator. Users are assigned into all female or 
all male groups. They are allowed to login using given id (refer Fig. 1) and participate in learning sessions. 
Instruction to use Collab21earn DB is displayed prior authorized login (refer Fig. 2). Each user has the ability 
give comments and suggestions related to workplace scenarios displayed on the interface. End users have the 
authority to give feedback through 'I Agree' or T Disagree' buttons. The buttons are created in two types: 
iconic for high rich and textual for low rich. Feedback to other users in the same group is allowed. 
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Communication type in Collab21earn DB is restricted by their synchronicity: synchronous/ real time 
communication and asynchronous/ not real time communication. Synchronous sessions are constraint within 
60 minutes of learning sessions while asynchronous sessions are constraint by 7 days/168 hours, (refer Fig. 3 
and 4). Time is displayed in counted down format and if the time is over, time out notification appears and 
request users to continue with questionnaire session (refer Fig. 5). Users are allowed to continue with 
questionnaire if they have finished discussing all the scenarios. 

Thank you notifications will appear after finish answering all the questions and users will log out to end their 
learning session (refer Fig. 6). 



COLLAB2LEARN DISCUSSION BOARD 


Login 








Login 








Figure 1 . Login interface 



COLLAB2LEARN DISCUSSION BOARD 

Logout 



Instructions 

The scenarios that will be discussed are examples of problems or issues occur 
in the workplace. The scenarios might be related with positive personalities 
development, communications skills, ethics/ rules and regulations or work 
life exposure. The solutions and comments during this discussion will be 
evaluated for their confidence and quality. There will be no right or wrong 
solutions. Please response to the questions as honestly as you can. All 
responses will remain confidential. Only members of the research will have 
the access to the information you give. Thank you. 



I Agree 



Figure 2. Instructions to participate in Collab21earn DB 



COLLAB2LEARN DISCUSSION BOARD 

37 minute(s) remaining Logout 




Type 2 

You are in your break walking towards 

Question: 

What is ? 


Next scenario 


Add comment 

1. I don't really agree with this 

Posted by: Ali on June 30, 2012 10:30:12 



Figure 3. Example for synchronous and iconic learning session in Collab21earn Discussion Board 
C. System Usability 

User interface is focused on the user interaction and experiences for products not only related to computer 
applications such as computer websites, computer appliances and software applications, but to any 
applications users interact with control or display [6]. Design on the user interface requires developer to plan 
user interaction that is simple and efficient. Good design on the user interface will contribute to greater user 
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C0LLAB2LEARN DISCUSSION BOARD 

4 days 37 minute(s) 34 seconds remaining Logout 



Type 2 

You are in your break walking towards . 

Question: 

What is ? 





Next scenario 






Add comment 



1 . I don' t really agree with this 

Posted by: AlionJune 30,2012 10:30:12 



Figure 4: Example for asynchronous and textual learning session in Collab21earn Discussion Board 



COLLAB2LEARN DISCUSSION BOARD 



Type 2 . 

You are in your break walking towards . 

Question: 

What is 



Add con 



Your time has ended. Redirecting 
you to questionnaire. Than k you. 



OK 



1. I don' t really agree with this 

Posted by: AlionJune 30,2012 10:30:12 



Figure 5: Time out notification to notify the time is over and questionnaire session will begins 



COLLAB2LEARN DISCUSSION BOARD 



Thank you for participating in Collab21earn Discussion Board. If you are 
interested with this research result, please contact us. Thank you. 




Figure 6: Thank you notification to users for their participation during learning session and questionnaire session 

acceptance, usable and accepted to the marketplace. The goal in good interface design is easy to understand, 
easy to use and easy to learn. 

Usable is described as the view of appropriateness of a product or a tool to achieve specified goals with 
effectiveness, efficiency and satisfactory to specified users [7]. Usability of a tool or a system is unique at its 
own context or situation [8 and 9] such as for industrial context, educational context or everyday-general 
household context. A usable system is useful and easy to learn contains the right content and meet the 
specified requirements and user's satisfaction [10]. 

Designing a usable system is an iterative process from the users scope, user testing phases (simulations and 
prototype, measures performance), developers modified the system based on user testing measurements and 
finally integrates all system's usable functions in parallel and specified goal [10] identified by system 
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developers. Modifies the system and making changes according to testing results is an iterative process until 
the systems meets the requirements [11]. Gould (2004) described a reliable system with appropriate response 
time and systems that work to the specified functions with suitable user interface are aspects to cater during 
developing and designing a system. 

Easy to learn and use with well integrated functionalities are essential for a system to be usable [6, 12 and 
13]. System's manual or outreach/ training programs are important to support end users learnability. A 
satisfactory usable system contains workable specified functions, easy to learn and use at appropriate 
response time. 

D. System Usability Scale 

Usability test is important to test ease of use and learnability of a system. A usable system or a tool is viewed 
from its appropriateness to the context or situation [8] and achieved specified goals with effectiveness, 
efficiency and satisfactory to specified users [7]. The context for a system can be varied from industrial 
context, educational context, banking context or general context [9, 13, 14 and 15]. In other words; where the 
system would be used will determine either the system is usable for that specific context or not meeting the 
requirements. 

System Usability Scale (SUS) was developed in 1986 by John Brooke as an alternative instrument to measure 
system's usability suitable to its environment. The ten -item questionnaires were designed based on global 
view of subjective assessments of usability [12 and 15]. The scale measure ease of use for a system and using 
the same scale, other related system from other country is possible to be compared. Though SUS is often 
known as 'quick and dirty' scale, it has been widely used to various global context related to systems 
usability specifically computer system such as company websites, database application and computer 
programs that involved with users [8 and 12]. 

This study explored online discussion board usability within collaborative learning sessions. Two groups 
experimented the prototype at different feedback formats; textual and iconic. Both groups explored all 
functionalities including textboxes, buttons and icons. SUS questionnaires were distributed to all 
participants. Reliability analysis results and average scores between both tests were explored to satisfy 
authors' enquiry with lower reliability results compared to other study using the same scale [7, 9, 14 and 15]. 

III. Method 

A. Research Design 

This research was design for experimental study on selected participants. A developed prototype called 
Collab21earn Discussion Board was used as the external tool for collaborative learning sessions. Several 
workplace scenarios were designed as the learning content. Participants communicate and discuss within 
their group on questions given based from the scenarios. Analyses were conducted from electronic 
questionnaire completed by participants at the end of learning session or their discussions. 

B. Research Objective 

The research objective for this study is "To test a prototype usability using SUS questionnaire at different 
sessions based on the prototype's functionalities". 

C. Sampling 

Total of 57 participants were selected from an institute of higher learning. All participants had basic 
knowledge of using online discussion board. Participants were assigned to groups with different prototype 
functionalities. 28 participants experimented textual feedback format and 29 participants experimented iconic 
feedback format. 

D. Experimental Procedure 

Experimental procedure for Collab21earn usability testing was conducted for independent participants in 
different sessions. Participants were divided into two types of groups (textual feedback format and iconic 
feedback format). Each participants will independently login to the same online discussion board 
(http://www.collab21earn.net) using their given ID. 

All participants were required to explore prototype functionalities according to its function. Participants were 
divided into groups to experiment Collab21earn DB at different feedback formats (textual or iconic). 
Usability test was conducted to the Collab21earn DB usability to 57 participants and was measured by SUS 
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scale. Independent participants were selected and measuring instruments were given at the end of discussion 
sessions. 

Selected participants log-in to Collab21earn Discussion Board web site using their given ID. Workplace 
scenarios were displayed and participants were allowed to give comments and click button 'Add Comment' . 
All comments were displayed below the comment's textbox. Participants were allowed to 'Agree' or 
'Disagree' their group members' comments excluding their own. Buttons 'Agree' and 'Disagree' were 
displayed either in textual or iconic format (refer Fig. 7 A and 7B) 

Nicknames were assigned to each participant and they are allowed to give comments more than once for each 
scenario. Participants were allowed to answer the questionnaire either if they are tested all functionalities or 
if the time allocated was up. At the end of testing session, they were instructed to test and explore all 
functionalities including textboxes, buttons and icons. At the end of testing sessions, all participants will 
answer a 10-items questionnaire to measure Collab21earn usability. 




Figure 7(A). Textual feedback format button. 
Figure 7(B). Iconic feedback format button. 

E. Questionnaire Design 

Electronic questionnaire will automatically generate at the end of learning session for each participants. 
Participants are allowed to answer several times, but only the latest answer will be used for further analysis. 
Questionnaire designed to measure dependent variables for the experimental study. 6-point Likert scale (1- 
Really disagree, 2- Mostly disagree, 3- Disagree a little, 4- Agree a little, 5- Mostly agree and 6- Really 
agree) contained 10-items System Usability Scale [8, 12, 15 and 16] was used to measure Collab21earn 
usability. Both test consisted five of items which are positive and five items that are negatively worded. Item 
1 to Item 5 are positive worded and Item 6 to Item 10 are negative worded. Examples of the items are as 
follows: 

Item 1: "I think I would like to use this system frequently 9 '. 

Item 2: "I thought the system was easy to use ". 

Item 6: "I found the system unnecessarily complex". 

Item 8: "I thought there were too much inconsistency in this system". 

F. Reliability Analysis 

Reliability for internal correlations between 10-items was calculated based on Cronbach's Alpha 
measurement. Internal consistency for usability calculated acceptable reliability of a = 0.757 [15]. Higher 
results reliability of a =0.91 were discovered by previous researchers [8 and 12]. Dillenbourg et al. (2008) 
described participants in groups contributed to biasness during solving tasks however [15] described items 
internal consistency of 0.7 and above is acceptable. 



IV. Result and Discussions 

The research objective for this study is 'To test a prototype usability using SUS questionnaire at different 
sessions based on the prototype's functionalities'. The usability of the prototype was explored. Analysis 
comparing with average scores in 10-items SUS for Collab21earn DB for different feedback format (N=57; 
Textual=28 and Iconic= 29) were held using SPSS 13.0 [17]. 

Almost 85% participants agree to use Collab21earn frequently, however approximately; only 12% 
participants disagree a little with the thought of using Collab21earn frequently. This has shown, most 
participants agreed to use Collab21earn frequently with small number of disagreement from the participants. 
Textual feedback format carried lower average (Mean= 4.18, SD= 1.188, Median= 4.00) compared to iconic 
feedback format (M= 4.55, SD= 0.870, Median= 5.00). Higher participants from iconic feedback agreed to 
use Collab21earn frequently compared with textual feedback format (refer Fig. 8A). 

The result has shown that 37% participants agree the most and 33% participants strongly agree with the 
thought Collab21earn was easy to use. This reflected that the majority agreed Collab21earn was easy to use 
with only 12% disagree Collab21earn was easy to use. Similar average scores between textual (Mean= 4.86, 
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SD= 1.177, Median= 5.00) and iconic (Mean= 4.83, SD= 1.197, Median= 5.00) feedbacks formats with the 
agreement that Collab21earn was easy to use (refer Fig. 8B). 




Strongly disagree Mostly cisagree Disagree, a little Agree, a little Mostly agree Strongly agree 

Item 1: 1 think I would like to use this system frequently 



8(A) 




Strongly disagree Mostly cisacree Dsacree, a little Agree, a little Mostly agree Strongly agree 

Item 2: 1 thought the system was easy to use 



8(B) 



Figure 8A. Collab21earn result for frequent usage. 
Figure 8B. Colalb21earn result for ease of use 

Approximately 80% participants found that the functions in Collab21earn were well integrated. Only 20% 
participants found the functions were not really well integrated. Textual feedback format (Mean= 4.32, SD= 
0.983, Median= 4.00) carried lower agreement with Collab21earn functionalities integration compared to 
iconic feedback format (Mean= 4.38, SD= 1.015, Median= 5.00) (refer Fig. 9A). 

86% participants agree that most people would learn to use this system very quickly, however approximately 
5% participants strongly and mostly imagine that Collab21earn was not easy to learn. This reflected 
Collab21earn is easy to learn for majority of participants. Participants in iconic feedback format (Mean= 4.72, 
SD= 1.131, Median= 5.00) agreed more that most people would learn to use Collab21earn very quickly 
compared to textual feedback format (Mean= 4.54, SD= 1.071, Median= 5.00) (refer Fig. 9B). 
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Mostly disagree Disagree, a little Agree, a little Mostly agree Strongly agree 

Item 3: 1 found the various functions in this system were well integrated 

9(A) 



XX 



Strongly disagree Mostly disagree Disagree, a little 



a little Mostly agree Strongly agree 



Item 4: 1 would imagine that most people would learn to use this system very 
quickly 



9(B) 



Figure 9A. Collab21earn result for functionality integration 
Figure 9B. Collab21earn result for system's learnability to most people 

Only 14% participants not really feel very confident to use Collab21earn. 86% participants felt very confident 
using Collab21earn. None from both iconic and feedback format don't feel very confident using Collab21earn. 
Textual feedback format (Mean= 4.68, SD= 0.983, Median= 5.00) carried lower scores to use Collab21earn 
confidently compared to iconic feedback format (Mean= 4.83, SD= 1.071, Median= 5.00) (refer Fig. 10A). 
Approximately 89% participants agree with the system functionalities was complex unnecessarily. Only 11% 
does not found Collab21earn was not complex. Textual (Mean= 4.54, SD= 1.138, Median= 4.50) feedback 
format found that Collab21earn was unnecessarily complex more than iconic (Mean= 4.38, SD= 0.979 
Median= 4.00) feedback format (refer Fig. 10B). 
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Observation during testing session discovered several participants were biased in scores. Contradict results 
were discovered for system's consistency; Item 3 & 8, easy to use; Item 2 & 7 and system's learnability; Item 
4 & 10. It was discovered that participants chose higher score points for all items even though item 2, 3 and 4 
are positive worded while item 7, 8 and 10 are negative worded questions. They were influenced by others 
scores especially when participated in group discussions among participants attitude [1]. Students were 
observed to discuss quietly among themselves during answering the questionnaire. This reflected research 
conducted in group discussions that requires them to answer questionnaires individually should be monitor 
strictly by the instructors. 



Disagree, a little Agree, a little Mostly agree Strongly agree 

Item 5: 1 felt very confident using the system 



10(A) 




Strongly disagree Mostly disagree □sagree, a little Agree, a little Mostly agree Strongly aagree 

Item 6: 1 found the system unnecessarily complex 



410B) 



Figure 10A. Collab21earn result for confident to use 
Figure 10B. Collab21earn result for system's complexity 

Approximately 89% participants thought they need support from technical person to be able to use this 
system. Higher number of iconic (Mean= 4.83, SD= 0.889, Median= 5.00) feedback format agreed that 
technical person was needed compared to textual (Mean= 4.50, SD= 1.319, Median= 4.00) feedback format 
(refer Fig. 11 A). 
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Approximately 70% participants thought there were too much inconsistency in Collab21earn, however, only 
5% participants strongly disagree with Collab21earn's inconsistency. Both textual (Mean= 4.21, SD= 1.197, 
Median= 4.00) and iconic (Mean= 4.10, SD= 1.448, Median= 4.00) feedback format have similar agreement 
with the thought there was too much inconsistency in Collab21earn (refer Fig. 11B). 

Interrupted internet connection during testing session has contributed to inconsistency and feeling of 
awkwardness (refer Fig. 8 A) pertaining lack of response time [7 & 9]. Appropriate response time was 
important for a system to be usable. The interrupted mood contributed to participants' judgment that 
Collab21earn was inconsistent and complex though the functions were well integrated and easy to use by 
some participants [10]. 



Strongly disagree Mostly disagree Disagree, a little Agree, a little Mostly agree Strongly agree 

Item 7: 1 think that I would need a support of a technical person to be able to 
use this system 



11 A) 




Strongly disagree Mostly disagree Dsagree, a little Agree, a little Mostly agree Strongly agree 

Item 8: 1 thought there were too much inconsistency in this system 

11(B) 



Figure 11 A. Collab21earn result for technical support 
Figure 11B. Collab21earn result for system's inconsistency 
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Approximately 25% participants found the system was not awkward to be used, however 75% participants 
found that Collab21earn was very awkward to be used. Higher scores from textual (Mean= 4.39, SD= 1.031, 
Median= 4.00) feedback format thought Collab21earn was awkward to use compared from iconic (Mean= 
4.10, SD= 1.398, Median= 4.00) feedback format (refer Fig. 12A). 

Only 12% participants mostly and strongly disagree that they need to learn a lot before using Collab21earn. 
More than 40% mostly and strongly agreed that they needed to learn a lot of things before using 
Collab21earn. Higher number from iconic (Mean= 4.45, SD= 1.378, Median= 5.00) feedback format agreed 
that they needed to learn a lot before using Collab21earn compared to textual (Mean= 3.98, SD= 1.499, 
Median= 4.00) feedback format disagreed that they needed to learn a lot before using Collab21earn (refer Fig. 
12B). 



Strongly disagree Mostly disagree Disagree, a little Agree, a little Mostly agree Strongly a 

Item 9: 1 found the system very cumbersome /"awkward to use 

12(A) 



Strongly disagree Mostly disagree Disagree, a little Agree, a little Mostly agree Strongly agree 

Item 10: 1 needed to learn a lot of things before I could get going with this 
system 



12(B) 



Figure 12 A. Collab21earn result for system's usage awkwardness 
Figure 12B. Collab21earn result for system's learnability before it is able to use 
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V. Conclusions 

Collab21earn Discussion Board generally is usable as communication platform whereby students can 
participate in collaborative learning, give comments and feedback to their group members. Overall, 
Collab21earn carried positive scores (4.00 and above) to both textual and iconic feedback formats. Both 
textual and iconic feedback formats scored were reliable as the mean and median scores were not much 
different [17]. 

Technical improvements need to be done especially on the system's accesses and multiple comments 
submission during offline period. Ability to access systems' contents and functionality; and 'keyed -in' 
comments are saved even during the offline period will certainly improve system's usability. Technical issue 
observed during the testing session influenced scores to the ease of use and system consistency even though 
they are confident and learn to use Collab21earn very quickly. Leavitt & Shneiderman (2006) in Research- 
based Web Design and User Guideline described iterative process in testing and modifies based on the 
testing results will increased product efficiency thus prevent mistakes and cost effective. 

A. Future Works 

Currently, Colllab21earn can be accessed through network and it is constrained to the internet connectivity. 
The 'downtime' or 'offline' contributed to low usability measurements. Authors need to improve with 
Collab21earn functionality because users are required to submit comments several times when the network is 
down. It is suggested to improve with the installation or technical distributions of Collab21aern for users. 
Management issues during conducting testing sessions and instructing users from beginning till questionnaire 
also need to be improved. Instructions for students' participating group discussion need to be clear and 
monitored by the instructors to lessen biasness attitude among participants. 
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